Systems and methods for remote operation of robot vehicles

ABSTRACT

An autonomous robot vehicle in accordance with aspects of the present disclosure includes a land vehicle conveyance system, a sensor system configured to capture information including surrounding environment information and/or vehicle subsystem information, a communication system configured to communicate with a remote human operator management system, at least one processor, and a memory storing instructions. The instructions, when executed by the processor(s), cause the autonomous robot land vehicle to, autonomously, determine based on the captured information to request a remote human operator, and communicate a request to the remote human operator management system for a remote human operator to assume control of the land vehicle conveyance system, where the request includes at least a portion of the captured information.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. Provisional ApplicationNo. 62/538,538, filed on Jul. 28, 2017, which is hereby incorporated byreference in its entirety.

FIELD OF THE TECHNOLOGY

The present application relates to autonomous vehicles, and inparticular, to remote operation of autonomous vehicles by humanoperators.

BACKGROUND

The field of fully-autonomous and/or semi-autonomous robots is a growingfield of innovation. Robots are being used for many purposes includingwarehouse inventory operations, household vacuuming robots, hospitaldelivery robots, sanitation robots, and military or defenseapplications.

In the consumer space, handling and delivery of goods and services byautonomous vehicles could improve society in many ways. For example,rather than spending time traveling to a merchant, a person can insteadengage in productive work while waiting for an autonomous vehicle todeliver the goods and/or services. With fewer vehicles on the road,traffic conditions would also improve. For example, instead of severalpeople traveling to merchants in several vehicles, a single autonomousvehicle could deliver goods and/or services to those people and therebyreduce the number of vehicles on the road. Other uses and applicationsfor autonomous vehicles are possible as technology progresses.Accordingly, there is interest in developing technologies for autonomousvehicles.

SUMMARY

This disclosure relates to a fully-autonomous and/or semi-autonomousrobot fleet and, in particular, to a fleet of robot vehicles in eitherunstructured outdoor environment or closed environments. In one aspect,the present disclosure provides systems and method for remote operationof autonomous vehicles by human operators, and for prioritizing certainautonomous vehicles for remote human operation based on various factors.In various embodiments, the autonomous vehicles can be any land vehicle,including land vehicles configured to carry persons, cargo, goods, orother objects or substances.

In accordance with aspects of the present disclosure, an autonomousrobot land vehicle includes a land vehicle conveyance system, a sensorsystem configured to capture information including surroundingenvironment information and/or vehicle subsystem information, acommunication system configured to communicate with a remote humanoperator management system, one or more processors, and a memory storinginstructions. The instructions, when executed by the one or moreprocessors, cause the autonomous robot land vehicle to, autonomously,determine based on the captured information to request a remote humanoperator, and communicate a request to the remote human operatormanagement system for a remote human operator to assume control of theland vehicle conveyance system, where the request includes at least aportion of the captured information.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the processor(s), cause theautonomous robot land vehicle to determine, based on the vehiclesubsystem information, that at least one subsystem is not functioningproperly.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thecaptured information, that a next maneuver cannot be autonomouslydetermined.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thesurrounding environment information, that a surrounding environmentsituation is not recognized.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to access predeterminedsituations or predetermined roadways that are predetermined to require arequest for a remote human operator, and determine, based on thecaptured information, that at least one of the predetermined situationsor the predetermined roadways has been encountered

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thesurrounding environment information, that a surrounding environmentsituation should not be autonomously handled.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thevehicle subsystem information, that a situation of at least onesubsystem is not recognized.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to access a predeterminedoperating geography, and determine, based on the surrounding environmentinformation, that the autonomous robot land vehicle is outside thepredetermined operating geography.

In various embodiments, in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thecaptured information, that multiple attempts to execute an autonomousmaneuver have failed.

In various embodiments, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to determine arisk rating based on the captured information.

In various embodiments, in determining a risk rating based on thecaptured information, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to determine therisk rating based on estimated safety of autonomous risk mitigationprocedures that can be effectuated while waiting for a remote humanoperator to assume control.

In various embodiments, in determining the risk rating based onestimated safety of autonomous risk mitigation procedures, theinstructions, when executed by the at least one processor, cause theautonomous robot land vehicle to access the risk mitigation procedures,estimate based on the captured information probability of success inexecuting the risk mitigation procedures, and determine the risk ratingbased on the estimated probability of success.

In various embodiments, in estimating the probability of success inexecuting the risk mitigation procedures, the instructions, whenexecuted by the at least one processor, cause the autonomous robot landvehicle to estimate the probability of success based on at least one of:road speed limits, current vehicle speed, current speed of surroundingvehicles, number of surrounding vehicles, number of surroundingpedestrians, number of surrounding objects, road width, weatherconditions, available time to react, or proximity to surroundingvehicles. Other factors are contemplated to be within the scope of thepresent disclosure.

In various embodiments, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to control theland vehicle conveyance system to mitigate risk while waiting for aremote human operator to assume control.

In various embodiments, in controlling the land vehicle conveyancesystem to mitigate risk, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to perform atleast one of: park at a curb, parking in a parking lot, travel onroadways in search of parking, or depart from a planned route to aroadway having lower risk.

In various embodiments, in controlling the land vehicle conveyancesystem to mitigate risk, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to travel at aslower speed than a travel speed without the risk.

In various embodiments, the autonomous robot land vehicle does notcontain any space for a human passenger or human operator to be withinthe autonomous robot land vehicle.

In aspects of the present disclosure, a system for assigning remotehuman operators to autonomous land vehicles includes a databaseincluding experience information on a plurality of remote humanoperators, a communication system configured to communicate with remoteautonomous land vehicles, one or more processors, and a memory storinginstructions. The instructions, when executed by the processor(s), causethe system to receive via the communication system requests from theremote autonomous land vehicles for remote human operators to assumecontrol, where each of the requests includes risk information generatedby the corresponding remote autonomous land vehicle, determine that thenumber of the requests is greater than the number of available remotehuman operators, access risk ratings associated with the requests wherethe risk ratings include at least one of ratings determined by thesystem based on the risk information generated by the remote autonomousland vehicles or ratings that are determined by the remote autonomousland vehicles and included in the risk information generated by theremote autonomous land vehicles, and assign at least some of theavailable remote human operators to at least some of the remoteautonomous land vehicles based on at least one of the risk ratings orthe experience information on the remote human operators.

In various embodiments, in assigning at least some of the availableremote human operators to at least some of the remote autonomous landvehicles, the instructions, when executed by the processor(s), cause thesystem to prioritize assignment of the available remote human operatorsto remote autonomous land vehicles having highest risk ratings.

In various embodiments, the experience information includes an amount ofexperience, and in prioritizing assignment of the available remote humanoperators to remote autonomous land vehicles having highest riskratings, the instructions, when executed by the at least one processor,cause the system to prefer available remote human operators who havehigher amounts of experience.

In various embodiments, the experience information includes experiencedriving in particular regions, and in assigning at least some of theavailable remote human operators to at least some of the remoteautonomous land vehicles, the instructions, when executed by theprocessor(s), cause the system to determine, based on the experienceinformation, a group of the available remote human operators who haveexperience driving in a particular region in which one of the remoteautonomous land vehicles is located, and assign one of the remote humanoperators from the group to the one remote autonomous land vehicle.

In various embodiments, the experience information includes experiencedriving in particular situations, and each of the requests includes anindication of a risk situation.

In various embodiments, the risk situation includes a vehicle subsystemnot functioning properly, and in assigning at least some of theavailable remote human operators to at least some of the remoteautonomous land vehicles, the instructions, when executed by theprocessor(s), cause the system to determine, based on the experienceinformation, a group of the available remote human operators who haveexperience remotely operating a vehicle with the vehicle subsystem notfunctioning properly, and assign one of the remote human operators fromthe group to the one of the remote autonomous land vehicles. In variousembodiments, the vehicle subsystem that is not functioning properly is anavigation subsystem.

In various embodiments, the risk situation includes a surroundingenvironment situation, and in assigning at least some of the availableremote human operators to at least some of the remote autonomous landvehicles, the instructions, when executed by the at least one processor,cause the system to determine, based on the experience information, agroup of the available remote human operators who have experienceremotely operating a vehicle in the surrounding environment situation,and assign one of the remote human operators from the group to the oneof the remote autonomous land vehicles. In various embodiments, thesurrounding environment situation includes one or more of anunrecognized object, an emergency services vehicle signaling anemergency, a human directing traffic, and/or an excessive number ofpedestrians or vehicles.

Further details and aspects of exemplary embodiments of the presentdisclosure are described in more detail below with reference to theappended figures.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the features and advantages of the disclosedtechnology will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of the technology are utilized, and the accompanying drawingsof which:

FIG. 1 is an exemplary view an autonomous robot fleet, wherein eachvehicle within a fleet or sub-fleet can be branded for an entity;

FIG. 2 is an exemplary ISO view of a robot vehicle, part of anautonomous robot fleet, illustrating securable compartments within thevehicle;

FIG. 3 is an exemplary front view of a robot vehicle, part of anautonomous robot fleet, shown in comparison to the height of an averageperson;

FIG. 4 is an exemplary right side view of a robot vehicle, part of anautonomous robot fleet, illustrating a configuration with two large sidedoors, each enclosing securable compartments;

FIG. 5 is an exemplary left side view of a robot vehicle, part of anautonomous robot fleet, shown in comparison to the height of an averageperson;

FIG. 6 is an exemplary rear view of a robot vehicle, part of anautonomous robot fleet;

FIG. 7 is an exemplary ISO view of a robot vehicle, part of anautonomous robot fleet, illustrating an autonomous lunch deliveryvehicle for any branded company;

FIG. 8 is an exemplary ISO view of a robot vehicle, part of anautonomous robot fleet, illustrating an autonomous pizza deliveryvehicle for any branded company;

FIG. 9 is an exemplary ISO view of a robot vehicle, part of anautonomous robot fleet, illustrating an autonomous coffee deliveryvehicle for any branded company;

FIG. 10 is an exemplary ISO view of a robot vehicle, part of anautonomous robot fleet, illustrating an autonomous evening/nighttimedelivery vehicle for any branded company, comprising a lighted interior;

FIG. 11 is an exemplary flowchart representation of the logic for afleet management control module associated with a central server for therobot fleet;

FIG. 12 is an exemplary flowchart representation of the logic flow fromthe Fleet Management Control Module through the robot processor to thevarious systems and modules of the robot;

FIG. 13 is a diagram of an exemplary remote human operator system;

FIG. 14 is a diagram of an exemplary visual display of the remote humanoperator system of FIG. 13;

FIG. 15 is a diagram of an exemplary configuration of visualizing anautonomous land vehicle by an autonomous aerial vehicle;

FIG. 16 is a flow chart of an exemplary operation of an autonomousvehicle for requesting a remote human operator to assume control; and

FIG. 17 is a flow chart of an exemplary operation of a remote humanoperator management system for assigning remote human operators toautonomous vehicles.

DETAILED DESCRIPTION

This disclosure relates to a fully-autonomous and/or semi-autonomousrobot fleet and, in particular, to robot vehicles for transporting orretrieving persons or deliveries in either open unstructured outdoorenvironments or closed environments. In one aspect, the presentdisclosure provides systems and methods for remote operation ofautonomous vehicles by human operators, and for prioritizing certainautonomous vehicles for remote human operation based on various factors.Remote operation of a fully-autonomous or a semi-autonomous vehicle maybe appropriate in various situations. For example, if the autonomousvehicle is requested to travel to a destination that has not been fullymapped (e.g., large corporate or university campuses, or public parks,etc.), the autonomous vehicle may not be able to determine how to reachthe destination. In other scenarios, an autonomous vehicle may notrecognize a situation or may determine that it should not autonomouslyhandle a situation. Accordingly, the capability for a human operator toremotely operate an autonomous vehicle is a beneficial feature. However,as autonomous fleets grow, the number of vehicles will vastly outnumberthe number of remote human operators. In accordance with aspects of thepresent disclosure, systems and methods are disclosed for assigningremote human operators to autonomous vehicles based on various factors.

Provided herein is a robot fleet having robot vehicles operatingfully-autonomously or semi-autonomously and a fleet management modulefor coordination of the robot fleet, where each robot within the fleetis configured for transporting, delivering or retrieving goods orservices and is capable of operating in an unstructured open or closedenvironment. Each robot can include a power system, a conveyance system,a navigation module, at least one securable compartment or multiplesecurable compartments to hold goods, a controller configurable toassociate each of the securable compartments to an assignable customer acustomer group within a marketplace, or provider and provide entry whenauthorized, a communication module and a processor configured to managethe conveyance system, the navigation module, the sensor system, thecommunication module and the controller.

As used herein, the term “autonomous” includes fully-autonomous,semi-autonomous, and any configuration in which a vehicle can operate ina controlled manner for a period of time without human intervention.

As used herein, the term “fleet,” “sub-fleet,” and like terms are usedto indicate a number of land vehicles operating together or under thesame ownership. In some embodiments the fleet or sub-fleet is engaged inthe same activity. In some embodiments, the fleet or sub-fleet areengaged in similar activities. In some embodiments, the fleet orsub-fleet are engaged in different activities.

As used herein, the term “robot,” “robot vehicle,” “robot fleet,”“vehicle,” “all-terrain vehicle,” and like terms are used to indicate amobile machine that transports persons, cargo, items, and/or goods.Typical vehicles include cars, wagons, vans, unmanned motor vehicles(e.g., tricycles, trucks, trailers, buses, etc.), and unmanned railedvehicles (e.g., trains, trams, etc.), among other types of landvehicles.

As used herein, the term “user,” “operator,” “fleet operator,” and liketerms are used to indicate the entity that owns or is responsible formanaging and operating the robot fleet.

As used herein, the term “customer” and like terms are used to indicatethe entity that requests the services provided the robot fleet.

As used herein, the term “provider,” “business,” “vendor,” “third partyvendor,” and like terms are used to indicate an entity that works inconcert with the fleet owner or operator to utilize the services of therobot fleet to deliver the provider's product from and or return theprovider's product to the provider's place of business or staginglocation.

As used herein, the term “server,” “computer server,” “central server,”“main server,” and like terms are used to indicate a computer or deviceon a network that manages the fleet resources, namely the robotvehicles.

As used herein, the term “controller” and like terms are used toindicate a device that controls the transfer of data from a computer toa peripheral device and vice versa. For example, disk drives, displayscreens, keyboards, and printers all require controllers. In personalcomputers, the controllers are often single chips. As used herein thecontroller is commonly used for managing access to components of therobot such as the securable compartments.

As used herein a “mesh network” is a network topology in which each noderelays data for the network. All mesh nodes cooperate in thedistribution of data in the network. It can be applied to both wired andwireless networks. Wireless mesh networks can be considered a type of“Wireless ad hoc” network. Thus, wireless mesh networks are closelyrelated to Mobile ad hoc networks (MANETs). Although MANETs are notrestricted to a specific mesh network topology, Wireless ad hoc networksor MANETs can take any form of network topology. Mesh networks can relaymessages using either a flooding technique or a routing technique. Withrouting, the message is propagated along a path by hopping from node tonode until it reaches its destination. To ensure that all its paths areavailable, the network must allow for continuous connections and mustreconfigure itself around broken paths, using self-healing algorithmssuch as Shortest Path Bridging. Self-healing allows a routing-basednetwork to operate when a node breaks down or when a connection becomesunreliable. As a result, the network is typically quite reliable, asthere is often more than one path between a source and a destination inthe network. This concept can also apply to wired networks and tosoftware interaction. A mesh network whose nodes are all connected toeach other is a fully connected network.

As used herein, the term “module” and like terms are used to indicate aself-contained hardware component of the central server, which in turnincludes software modules. In software, a module is a part of a program.Programs are composed of one or more independently developed modulesthat are not combined until the program is linked. A single module cancontain one or several routines, or sections of programs that perform aparticular task. As used herein the fleet management module includessoftware modules for managing various aspects and functions of the robotfleet.

As used herein, the term “processor,” “digital processing device” andlike terms are used to indicate a microprocessor or central processingunit (CPU). The CPU is the electronic circuitry within a computer thatcarries out the instructions of a computer program by performing thebasic arithmetic, logical, control and input/output (I/O) operationsspecified by the instructions.

In accordance with the description herein, suitable digital processingdevices include, by way of non-limiting examples, server computers,desktop computers, laptop computers, notebook computers, sub-notebookcomputers, netbook computers, netpad computers, set-top computers,handheld computers, Internet appliances, mobile smartphones, tabletcomputers, personal digital assistants, video game consoles, andvehicles. Those of skill in the art will recognize that many smartphonesare suitable for use in the system described herein. Suitable tabletcomputers include those with booklet, slate, and convertibleconfigurations, known to those of skill in the art.

In some embodiments, the digital processing device includes an operatingsystem configured to perform executable instructions. The operatingsystem is, for example, software, including programs and data, whichmanages the device's hardware and provides services for execution ofapplications. Those of skill in the art will recognize that suitableserver operating systems include, by way of non-limiting examples,FreeBSD, OpenBSD, NetBSD®, Linux, Apple® Mac OS X Server®, Oracle®Solaris®, Windows Server®, and Novell® NetWare®. Those of skill in theart will recognize that suitable personal computer operating systemsinclude, by way of non-limiting examples, Microsoft® Windows®, Apple®Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU/Linux®. Insome embodiments, the operating system is provided by cloud computing.Those of skill in the art will also recognize that suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia®Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google®Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS,Linux®, and Palm® WebOS®.

In some embodiments, the device includes a storage and/or memory device.The storage and/or memory device is one or more physical apparatus usedto store data or programs on a temporary or permanent basis. In someembodiments, the device is volatile memory and requires power tomaintain stored information. In some embodiments, the device isnon-volatile memory and retains stored information when the digitalprocessing device is not powered. In some embodiments, the non-volatilememory includes flash memory. In some embodiments, the non-volatilememory includes dynamic random-access memory (DRAM). In someembodiments, the non-volatile memory includes ferroelectric randomaccess memory (FRAM). In some embodiments, the non-volatile memoryincludes phase-change random access memory (PRAM). In some embodiments,the device is a storage device including, by way of non-limitingexamples, CD-ROMs, DVDs, flash memory devices, magnetic disk drives,magnetic tapes drives, optical disk drives, and cloud computing basedstorage. In some embodiments, the storage and/or memory device is acombination of devices such as those disclosed herein.

In some embodiments, the digital processing device includes a display tosend visual information to a user. In some embodiments, the display is acathode ray tube (CRT). In some embodiments, the display is a liquidcrystal display (LCD). In some embodiments, the display is a thin filmtransistor liquid crystal display (TFT-LCD). In some embodiments, thedisplay is an organic light emitting diode (OLED) display. In varioussome embodiments, on OLED display is a passive-matrix OLED (PMOLED) oractive-matrix OLED (AMOLED) display. In some embodiments, the display isa plasma display. In some embodiments, the display is a video projector.In some embodiments, the display is interactive (e.g., having a touchscreen or a sensor such as a camera, a 3D sensor, a LiDAR, a radar,etc.) that can detect user interactions/gestures/responses and the like.In still some embodiments, the display is a combination of devices suchas those disclosed herein.

The Fleet of Robot Vehicles

Provided herein is a robot fleet 100, as illustrated in FIG. 1, havingrobot vehicles 101, with each one operating fully-autonomously orsemi-autonomously.

As illustrated in FIGS. 3-6, one exemplary configuration of a robot 101is a vehicle configured for land travel, such as a smallfully-autonomous (or semi-autonomous) automobile. The exemplaryfully-autonomous (or semi-autonomous) automobile is narrow (i.e., 2-5feet wide), low mass and low center of gravity for stability, havingmultiple secure compartments assignable to one or more customers,retailers and/or vendors, and designed for moderate working speed ranges(i.e., 1.0-45.0 mph) to accommodate inner-city and residential drivingspeeds. Additionally, in some embodiments, the land vehicle robot unitsin the fleet are configured with a maximum speed range from 1.0 mph toabout 90.0 mph for high speed, intrastate or interstate driving. Eachrobot in the fleet is equipped with onboard sensors 170 (e.g., cameras(running at a high frame rate, akin to video), LiDAR, radar, ultrasonicsensors, microphones, etc.) and internal computer processing toconstantly determine where it can safely navigate, what other objectsare around each robot and what it may do.

In in some embodiments, the robot fleet is fully-autonomous.

In in some embodiments, the robot fleet is semi-autonomous. In someembodiments, it may be necessary to have human interaction between therobot 101, the fleet operator 200, the provider 204 and/or the customer202 to address previously unforeseen issues (e.g., a malfunction withthe navigation module; provider inventory issues; unanticipated trafficor road conditions; or direct customer interaction after the robotarrives at the customer location).

In in some embodiments, the robot fleet 100 is controlled directly bythe user 200. In some embodiments, it may be necessary to have directhuman interaction between the robot 101 and/or the fleet operator 200 toaddress maintenance issues such as mechanical failure, electricalfailure or a traffic accident. Aspects of the present disclosurerelating to remote operation of the robot vehicles by a human operatorwill be described in more detail in connection with FIGS. 13-17.

In some embodiments, the robot fleet is configured for land travel. Insome embodiments, each robot land vehicle in the fleet is configuredwith a working speed range from 13.0 mph to 45.0 mph. In someembodiments, the land vehicle robot units in the fleet are configuredwith a maximum speed range from 13.0 mph to about 90.0 mph.

In some embodiments of the robot fleet, the autonomous robots within thefleet are operated on behalf of third party vendor/service provider.

For example, a fleet management service is established to provide aroving delivery service for a third party beverage/food provider (e.g.,a coffee service/experience for a third party vendor (i.e., Starbucks)).It is conceived that the fleet management service would provide asub-fleet of “white label” vehicles carrying the logo and products ofthat third party beverage/food provider to operate eitherfully-autonomously or semi-autonomously to provide this service.

In some embodiments of the robot fleet, the autonomous robots within thefleet are further configured to be part of a sub-fleet of autonomousrobots, and each sub-fleet is configured to operate independently or intandem with multiple sub-fleets having two or more sub-fleets (100-a,100-b).

For example, a package delivery service is configured to offer multiplelevels of service such as “immediate dedicated rush service,”“guaranteed morning/afternoon delivery service,” or “general deliveryservice.” A service provider could then have a dedicated sub-fleet ofdelivery vehicles for each type of service within their overall fleet ofvehicles. In yet another example, a third party has priority over acertain number of vehicles in the fleet. In so doing, they can guaranteea certain level of responsiveness. When they aren't using the vehicles,the vehicles are used for general services within the fleet (e.g., otherthird parties).

In some embodiments, the robot fleet is controlled directly by the user.

In some embodiments, there will likely be times when a vehicle breaksdown, has an internal system or module failure or is in need ofmaintenance. For example, in the event that the navigation module shouldfail, each robot within the fleet is configurable to allow for directcontrol of the robot's processor to override the conveyance and sensorsystems (i.e., cameras, etc.) by a fleet operator to allow for the safereturn of the vehicle to a base station for repair.

The Operating Environments

In some embodiments, the unstructured open environment is a non-confinedgeographic region accessible by navigable pathways, including, forexample, public roads, private roads, bike paths, open fields, openpublic lands, open private lands, pedestrian walkways, lakes, rivers orstreams.

In some embodiments, the closed environment is a confined, enclosed orsemi-enclosed structure accessible by navigable pathways, including, forexample, open areas or rooms within commercial architecture, with orwithout structures or obstacles therein, airspace within open areas orrooms within commercial architecture, with or without structures orobstacles therein, public or dedicated aisles, hallways, tunnels, ramps,elevators, conveyors, or pedestrian walkways.

In some embodiments, the navigation module controls routing of theconveyance system of the robots in the fleet in the unstructured open orclosed environments.

The Fleet Management Module

In some embodiments of the robot fleet 100, the fleet includes a fleetmanagement module 120 (associated with a central server) forcoordination of the robot fleet 100 and assignment of tasks for eachrobot 101 in the fleet. The fleet management module coordinates theactivity and positioning of each robot in the fleet. In addition tocommunicating with the robot fleet, fleet owner/operator and/or user,the fleet management module also communicates withproviders/vendors/businesses and customers to optimize behavior of theentire system.

The fleet management module works in coordination with a central server110, typically located in a central operating facility owned or managedby the fleet owner 200.

As illustrated in FIG. 11, in one embodiment, a request is sent to amain server 110 (typically located at the fleet owner's or fleetmanager's location), which then communicates with the fleet managementmodule 120. The fleet management module then relays the request to theappropriate provider 204 of the service (e.g., restaurant, deliveryservice, vendor or retailer) and an appropriate robot or robots 101 inthe fleet. The best appropriate robot(s) in the fleet within thegeographic region and typically closest to the service provider, is thenassigned the task, and the provider of the service 204 then interactswith that robot 101 at their business (e.g., loading it with goods, ifneeded). The robot then travels to the customer 202 and the customerinteracts with the robot to retrieve their goods or service (e.g., thegoods ordered). An interaction can include requesting the robot to openits compartment 102, 104 through the customer's app or through a userinterface on the robot itself (using, e.g., RFID reader and customerphone, a touchpad, a keypad, voice commands, vision-based recognition ofthe person, etc.). Upon completion of the delivery (or retrieval, ifappropriate), the robot reports completion of the assignment and reportsback to the fleet management module for re-assignment.

As further illustrated in FIG. 12, and previously noted, in someembodiments, the fleet management module 120 handles coordination of therobot fleet 100 and assignment of tasks for each robot 101 in the fleet.The fleet management module coordinates the activity and positioning ofeach robot in the fleet. The fleet management module also communicateswith vendors/businesses 204 and customers 202 to optimize behavior ofentire system. It does this by utilizing the robot's processor 125 toprocess the various inputs and outputs from each of the robot's systemsand modules, including: the conveyance system 130, the power system 135,the navigation module 140, the sensor system 170, 175, the communicationmodule 160, and the controller 150, to effectively manage and coordinatethe various functions of each robot in the fleet.

In some embodiments, the robot may be requested for a pick-up of an item(e.g., a document) with the intent of delivery to another party. In thisscenario, the fleet management module would assign the robot to arriveat a given location, assign a securable compartment for receipt of theitem, confirm receipt from the first party to the fleet managementmodule, then proceed to the second location where an informed receivingparty would recover the item from the robot using an appropriate PIN orother recognition code to gain access to the secure compartment. Therobot would then reports completion of the assignment and report back tothe fleet management module for re-assignment.

Conveyance Systems

Each robot vehicle 101 in the fleet includes a conveyance system 130(e.g., a drive system with a propulsion engine, wheels, treads, wings,rotors, blowers, rockets, propellers, brakes, etc.).

As noted previously, the robot fleet is configurable for land travel.Typical vehicles include cars, wagons, vans, unmanned motor vehicles(e.g., tricycles, trucks, trailers, buses, etc.), and unmanned railedvehicles (e.g., trains, trams, etc.), among other types of landvehicles.

In one exemplary embodiment, a robot land vehicle 101 is configured witha traditional 4-wheeled automotive configuration comprising conventionalsteering and braking systems. The drive train is configurable forstandard 2-wheel drive or 4-wheel all-terrain traction drive. Thepropulsion system (engine) is configurable as a gas engine, a turbineengine, an electric motor and/or a hybrid gas/electric engine.Alternatively, the robot could be configured with an auxiliary solarpower system 135 to provide back-up emergency power or power for minorlow-power sub-systems.

Alternative configurations of components to a total drive system with apropulsion engine could include wheels, treads, rotors, brakes, etc.

The Power System

In some embodiments, each robot of the robot fleet is configured withone or more power sources, which include the power system 135 (e.g.,battery, solar, gasoline, propane, etc.).

Navigation Module

Each robot in the fleet further includes a navigation module 140 fornavigation in the unstructured open or closed environments (e.g.,digital maps, HD maps, GPS, etc.). In some embodiments, the fleet 100relies on maps generated by the user, operator, or fleet operator,specifically created to cover the intended environment where the robotis configured to operate. These maps would then be used for generalguidance of each robot in the fleet, which would augment thisunderstanding of the environment by using a variety of on-board sensorssuch as cameras, LiDAR, altimeters or radar to confirm its relativegeographic position and elevation.

In some embodiments, for navigation, the fleet of robots uses internalmaps to provide information about where they are going and the structureof the road environment (e.g., lanes, etc.) and combine this informationwith onboard sensors (e.g., cameras, LiDAR, radar, ultrasound,microphones, etc.) and internal computer processing to constantlydetermine where they can safely navigate, what other objects are aroundeach robot and what they may do. In still other embodiments, the fleetincorporates on-line maps to augment internal maps. This information isthen combined to determine a safe, robust trajectory for the robot tofollow and this is then executed by the low level actuators on therobot.

In some embodiments, the fleet relies on a global positioning system(GPS) that allows users to determine their exact location, velocity, andtime 24 hours a day, in all weather conditions, anywhere in the world.

In some embodiments, the fleet of robots will use a combination ofinternal maps, sensors and GPS systems to confirm its relativegeographic position and elevation.

In some embodiments, the autonomous fleet is strategically positionedthroughout a geographic region in anticipation of a known demand.

Over time, a user 200 and/or a vendor 204 can anticipate demand forrobot services by storing data concerning how many orders (and what typeof orders) are made at particular times of day from different areas ofthe region. This can be done for both source (e.g., restaurants, grocerystores, general businesses, etc.) and destination (e.g., customer, otherbusinesses, etc.). Then, for a specific current day and time, thisstored data is used to determine what the optimal location of the fleetis given the expected demand. More concretely, the fleet can bepositioned to be as close as possible to the expected source locations,anticipating these source locations will be the most likely new ordersto come into the system. Even more concretely, it is possible toestimate the number of orders from each possible source in the next hourand weight each source location by this number. Then one can positionthe fleet so that the fleet optimally covers the weighted locationsbased on these numbers.

In some embodiments of the robot fleet, the positioning of robots can becustomized based on: anticipated use, a pattern of historical behaviors,or specific goods being carried.

Sensor Systems

As noted previously, each robot is equipped with a sensor system 170,which includes at least a minimum number of onboard sensors (e.g.,cameras (for example, those running at a high frame rate akin to video),LiDAR, radar, ultrasonic sensors, microphones, etc.) and internalcomputer processing 125 to constantly determine where it can safelynavigate, what other objects are around each robot, and what it may dowithin its immediate surroundings.

In some embodiments, the robots of the robot fleet further includeconveyance system sensors 175 configured to: monitor drive mechanismperformance (e.g., the propulsion engine); monitor power system levels135 (e.g., battery, solar, gasoline, propane, etc.); or monitor drivetrain performance (e.g., transmission, tires, brakes, rotors, etc.).

Communications Module

Each robot in the fleet further includes a communication module 160configurable to receive, store and send data to the fleet managementmodule, to a user, to and from the fleet management module 120, and toand from the robots in the fleet 100. In some embodiments, the data isrelated to at least user interactions and the robot fleet interactions,including, for example, scheduled requests or orders, on-demand requestsor orders, or a need for self-positioning of the robot fleet based onanticipated demand within the unstructured open or closed environments.

In some embodiments, each robot in the fleet includes at least onecommunication module configurable to receive, store and transmit data,and to store that data to a memory device, for future data transfer ormanual download.

In some embodiments, each business 204 and customer 202 has their ownapp/interface to communicate with the fleet operator 200 (e.g., “Nurocustomer app” for customers on their phone, “Nuro vendor app” forbusinesses on a tablet or phone or their internal computer system,etc.).

In some embodiments, the communication to the user and the robots in thefleet, between the robots of the fleet, and between the user and therobots in the fleet, occurs via wireless transmission.

In some embodiments, the user's wireless transmission interactions andthe robot fleet wireless transmission interactions occur via mobileapplication transmitted by an electronic device and forwarded to thecommunication module via: a central server, a fleet management module,and/or a mesh network.

In some embodiments, one preferred method of communication is to usecellular communication between the fleet manager and fleet of robots,(e.g., 3G, 4G, 5G, or the like). Alternatively, the communicationbetween the fleet control module and the robots could occur viasatellite communication systems.

In some embodiments, a customer uses an app (either on a cellphone,laptop, tablet, computer or any interactive device) to request a service(e.g., an on-demand food order or for a mobile marketplace robot to cometo them).

In some embodiments, the electronic device includes: a phone, a personalmobile device, a personal digital assistant (PDA), a mainframe computer,a desktop computer, a laptop computer, a tablet computer, and/orwearable computing device such as a communication headset, smartglasses, a contact lens or lenses, a digital watch, a bracelet, a ring,jewelry, or a combination thereof.

In accordance with aspects of the present disclosure, the communicationmodule 160 of each robot vehicle can be configured to communicate with aremote human operator. For example, the communication module 160 cancommunicate environmental videos captured by cameras running at a highframe rate to a remote operator, to enable the remote human operator tovisualize the vehicle's surroundings. Further, the communication module160 can receive instructions from the remote human operator forcontrolling the conveyance system to move the robot vehicle. Furtheraspects of remotely operating an autonomous vehicle by a human operatorwill be described in more detail in connection with FIGS. 13-18.

Goods and Services

In some embodiments, the user includes a fleet manager, asub-contracting vendor, a service provider, a customer, a businessentity, an individual, or a third party.

In some embodiments, the services include: subscription services,prescription services, marketing services, advertising services,notification services, or requested, ordered or scheduled deliveryservices. In particular embodiments, the scheduled delivery servicesinclude, by way of example, special repeat deliveries such as groceries,prescriptions, drinks, mail, documents, etc.

In some embodiments, the services further include: the user receivingand returning the same or similar goods within the same interaction(e.g., signed documents), the user receiving one set of goods andreturning a different set of goods within the same interaction, (e.g.,product replacement/returns, groceries, merchandise, books, recording,videos, movies, payment transactions, etc.), a third party userproviding instruction and or authorization to a goods or serviceprovider to prepare, transport, deliver and/or retrieve goods to aprinciple user in a different location.

In some embodiments, the services further include: advertising services,land survey services, patrol services, monitoring services, trafficsurvey services, signage and signal survey services, architecturalbuilding or road infrastructure survey services.

In some embodiments, at least one robot is further configured to processor manufacture goods.

In some embodiments, the processed or manufactured goods include:beverages, with or without condiments (such as coffee, tea, carbonateddrinks, etc.); various fast foods; or microwavable foods.

In some embodiments, the robots within the fleet are equipped forfinancial transactions. These can be accomplished using knowntransaction methods such as debit/credit card readers or the like.

Securable Compartments

As illustrated in FIG. 2, robots in the fleet are each configured fortransporting, delivering or retrieving goods or services and are capableof operating in an unstructured open environment or closed environment.In some embodiments, the vehicle 101 is configured to travel practicallyanywhere that a small all-terrain vehicle could travel on land, whileproviding at least one and preferably two large storage compartments102, and more preferably, at least one large compartment 102 isconfigured with smaller internal secure compartments 104 of variableconfigurations to carry individual items that are to be delivered to, orneed to be retrieved from customers.

Alternately, in some embodiments, the vehicle could be configured forproviding at least one and preferably two large storage compartments,and more preferably, at least one large compartment is configured withsmaller internal secure compartments of variable configurations to carryindividual items that are to be delivered to, or need to be retrievedfrom customers.

Further still, in some embodiments, the vehicle could be configured forhover travel, providing at least one and preferably two large storagecompartments, and more preferably, at least one large compartment isconfigured with smaller internal secure compartments of variableconfigurations to carry individual items that are to be delivered to, orneed to be retrieved from customers.

Further still, in some embodiments, the vehicle could be configured foraerial drone or aerial hover travel, providing at least one andpreferably two large storage compartments, and more preferably, at leastone large compartment is configured with smaller internal securecompartments of variable configurations to carry individual items thatare to be delivered to, or need to be retrieved from customers.

As illustrated in FIGS. 7-10, in some embodiments, the securablecompartments are humidity and temperature controlled for, for example,hot goods, cold goods, wet goods, dry goods, or combinations or variantsthereof. Further still, as illustrated in FIGS. 8-10, the compartment(s)are configurable with various amenities, such as compartment lightingfor night deliveries and condiment dispensers.

In some embodiments, the securable compartments are configurable forvarious goods. Such configurations and goods include: bookshelves forbooks, thin drawers for documents, larger box-like drawers for packages,and sized compartments for vending machines, coffee makers, pizza ovensand dispensers.

In some embodiments, the securable compartments are variablyconfigurable based on: anticipated demands, patterns of behaviors, areaof service, or types of goods to be transported.

Further still, each robot includes securable compartments to hold saidgoods or items associated with said services, and a controller 150configurable to associate each one of the securable compartments 102,104 to an assignable customer 202 or provider 204 and provide entry whenauthorized, Each robot vehicle further includes at least one processorconfigured to manage the conveyance system, the navigation module, thesensor system, instructions from the fleet management module, thecommunication module, and the controller.

As described previously, each robot is configured with securablecompartments. Alternately, a robot is configurable to contain a set ofgoods or even a mobile marketplace (similar to a mini bar at a hotel).

When a robot is assigned to a customer 202, one or more of thecompartments 102, 104 is also assigned to that customer. Each of thelarge compartments 12 is secured separately and can securely transportgoods to a separate set of customers 202.

Upon arrival of the robot to the customer destination, the customer canthen open their respective compartment(s) by verifying their identitywith the robot. This can be done through a wide variety of approachescomprising, but not limited to:

-   -   1. The customers can be given a PIN (e.g., 4 digit number) when        they make their initial request/order. They can then enter this        pin at the robot using the robot touchscreen or a keypad.    -   2. The customers can verify themselves using their mobile phone        and an RFID reader on the robot.    -   3. The customers can verify themselves using their voice and a        personal keyword or key phrase they speak to the robot.    -   4. The customers can verify themselves through their face, a        government ID, or a business ID badge using cameras and facial        recognition or magnetic readers on the robot.    -   5. The customers can verify themselves using their mobile phone;        by pushing a button or predetermined code on their phone (and        the system could optionally detect the customer is near the        robot by using their GPS position from phone)

In various embodiments, the interior space of each robot vehicle isconfigured to fill the interior space with securable compartments, suchthat the robot vehicles do not include any interior space for a humanoperator to operate the autonomous vehicle from within the vehicle.Rather, as explained below in connection with FIGS. 13-17, a humanoperator can remotely operate the autonomous vehicle.

Controller(s) and Processor(s)

In some embodiments, each robot in the robot fleet is equipped with oneor more processors 125 capable of both high-level computing forprocessing as well as low-level safety-critical computing capacity forcontrolling the hardware. The at least one processor is configured tomanage the conveyance system, the navigation module, the sensor system,instructions from the fleet management module, the communication moduleand the controller.

Further still, in some embodiments, each robot in the robot fleet isequipped with a controller 150 configurable to associate each one of thesecurable compartments 102, 104 to an assignable customer 202 orprovider 204 and provide entry when authorized.

The following will now describe control and processing in connectionwith remote operation of the robot vehicle by a human operator. Asmentioned above, remote operation of a fully-autonomous or asemi-autonomous vehicle may be appropriate in various situations. Thecapability for a human operator to remotely operate an autonomousvehicle is beneficial even where the autonomous vehicle can be locallyoperated by a human operator. This capability becomes much moreimportant where the interior space of the autonomous vehicle isconfigured to maximize commercial carrying capacity and includes nospace for a human operator to locally operate the vehicle from withinthe autonomous vehicle.

In various embodiments, an autonomous vehicle in accordance with aspectsof the present disclosure includes an interior space to hold a humanoperator to locally control the autonomous vehicle, but the vehicle canalso be controlled remotely by a remote human operator. In variousembodiments, an autonomous vehicle in accordance with aspects of thepresent disclosure includes no interior space to hold a human operatorto locally control the autonomous vehicle. Rather, in accordance withaspects of the present disclosure, a human operator can remotely operatethe autonomous vehicle. Such a configuration provides uniqueconsiderations. In contrast to an existing configuration in which ahuman operator located in an autonomous vehicle can override theautonomous operation and take over manual operation to avoid a hazard,various embodiments of the present disclosure do not include space in anautonomous vehicle for a human operator. Aspects of the presentdisclosure provide systems and methods for an autonomous vehicle torequest a remote human operator and for a remote human operator to beassigned to the autonomous vehicle based on various factors.

Referring now to FIG. 12, and as described above herein, an autonomousvehicle includes a processor 125 and a controller 150 for controllingvarious systems and modules of the autonomous vehicle and includes acommunication module 160 for communicating with external systems. In oneaspect of the present disclosure, the communication module 160 cancommunicate with a remote human operator system, which can be part of orseparate from the fleet management module 120.

FIG. 13 shows an exemplary embodiment of a remote human operator system300, which includes communication/processing equipment 310 and a humanoperator station 320. The human operator station 320 can resemble adriver station in a typical automobile and can include a driver seat322, a steering wheel 324, acceleration and brake pedals 326, a gearshifter 328, and a visual interface 330. In the illustrated embodiment,the visual interface 330 is in the form of a virtual-reality (VR) oraugmented-reality (AR) headset. In various embodiments, the visualinterface can include one or more display screens, such as LED, LCD,and/or OLED display screens. In various embodiments, the human operatorstation 320 can be configured to have the approximate touch response ofan actual driver station in an automobile. For example, the steeringwheel 324 can be configured to have the touch response of power steeringin an automobile, and the pedals 326 can be configured to approximatethe resistance of pedals in an actual automobile.

The instruments 324-328 of the human operator station 320 can beconnected or coupled to communication/processing equipment 310, whichenables communication between the human operator station 320 and theautonomous vehicle. In the illustrated embodiment, the human operatorstation 320 is connected to the communication/processing equipment 310by physical cables. In various embodiments, the human operator station320 can be wirelessly coupled to the communication/processing equipment310 using technologies such as Bluetooth. In various embodiments, thehuman operator station 320 need not be directly connected to thecommunication/processing equipment 310 and can be coupled to thecommunication/processing equipment 310 through intermediate devicesand/or networks.

In various embodiments, the communication/processing equipment 310 canestablish communications using various communications technologies,including, for example, IEEE 802.11x (WiFi), cellular 3G/4G/5G, wiredcommunications, and/or other wired or wireless communication protocols.The communication/processing equipment 310 includes one or moreprocessors, memories, machine instructions, and/or hardware forprocessing visual information for display by the visual interface 330.Persons skilled in the field will recognize various ways ofcommunicating, processing, and displaying visual information.

The communication/processing equipment 310 also processes signals fromthe human operator station 320 and translates them into controlinstructions for controlling the autonomous vehicle, such as controlinstructions for controlling the conveyance system (130, FIG. 12) of theautonomous vehicle to perform travel. In this manner, when the humanoperator turn turns the steering wheel 324, the communication/processingequipment 310 sends corresponding control instructions to the autonomousvehicle to instruct the vehicle to turn. As another example, when thehuman operator accelerates or brakes using the pedals 326 of the humanoperator station 320, the communication/processing equipment 310 sendscorresponding control instructions to the autonomous vehicle to instructthe vehicle to accelerate or brake, respectively. The embodiments andconfigurations of FIG. 13 are exemplary, and other configurations andvariations are contemplated to be within the scope of the presentdisclosure. For example, where the autonomous vehicle is a rail vehicle,the remote human operator system 300 may have a human operator station320 that reflects the actual operator station on a typicalnon-autonomous vehicle of the same or similar type.

Referring also to FIG. 14, there is shown a diagram of an exemplaryvisual display, which can be displayed in a VR/AR headset or on adisplay screen, or otherwise. As described above herein, each robotvehicle is equipped with a sensor system which can include cameras, suchas for example, those running at a high frame rate akin to video, andother sensors, and include internal computer processing to determinewhat other objects are around each robot vehicle. In accordance withaspects of the present disclosure, this visual information captured bythe sensor system of the autonomous vehicle can be processed andcommunicated to the remote human operator system 300 for display on thevisual interface 330. In various embodiments, the visual display canpresent the autonomous vehicle's surrounding environment from the pointof view of the human operator, such that the displayed content turns asthe driver's head turns. In various embodiments, the human operatorstation 320 can include multiple display screens (not shown) thatsurround the human operator station, and the display screens cansimultaneously display the surrounding environment of the autonomousvehicle. Other configurations and variations are contemplated to bewithin the scope of the present disclosure.

For example, in various embodiments, the visual display can present abird's eye view (not shown) of the autonomous vehicle. Referring also toFIG. 15, in various embodiments, an autonomous aerial vehicle 410 can beassociated with one or more autonomous land vehicles 420 and can captureimages or videos of the autonomous land vehicle 420 from a bird's eyeperspective. The captured images or videos can be communicated to theremote human operator system 300 for display on a visual interface 330.In this manner, the bird's eye perspective allows the human operator toeasily visualize the surrounding environment of the autonomous vehicle420 to better operate the autonomous vehicle 420 remotely.

In various embodiments, the autonomous aerial vehicle 410 can fly at analtitude that is free of obstacles such as traffic lights or expresswayoverpasses, among other things. In various embodiments, the visualinformation provided by the autonomous aerial vehicle can be processedto zoom in on the surroundings of the autonomous land vehicle 420 todifferent degrees. In various embodiments, the autonomous aerial vehicle410 can track the movement of the autonomous land vehicle 420, but candisengage if it is unable to track the movement safely, such as when theautonomous land vehicle 420 enters a tunnel. In various embodiments,there may be a fleet of roving autonomous aerial vehicles 410 that candynamically associate an autonomous aerial vehicle 410 with a particularland vehicle 420. Such a dynamic association can be communicated to theremote human operator system 300, so that the remote human operatorsystem 300 can switch between the visual information provided by theautonomous land vehicle 420 and the visual information provided by theautonomous aerial vehicle 410, while still maintaining remote operationof the autonomous land vehicle 420.

With continuing reference to FIG. 14, the visual display can includevarious indicators that inform the human operator of certain conditions.As an example, the visual display can include a mode indicator 332 tospecify whether the autonomous vehicle is operating in autonomous modeor in remote operation mode. In the autonomous mode, the humanoperator's interactions with the human operator station 320 do notaffect the movement of the autonomous vehicle 420. In the remoteoperation mode, the human operator controls the movement of theautonomous vehicle 420 by interacting with the human operator station320. In various embodiments, the human operator station can include amechanism (not shown) for switching between autonomous mode and remoteoperation mode, such as a physical switch or a touch interface button, avoice-activated command, or another mechanism. The illustrated visualdisplay also includes a warning indicator that can alert the humanoperator to various conditions 334. In various embodiments, the warningindicator 334 can alert the human operator of a loss of communication ora weak or unstable connection between the human operator station 320 andthe autonomous vehicle 420. For example, the warning indicator 334 canappear if a loss of communication occurs for a period of time, such asthree seconds or another period of time. In various embodiments, thewarning indicator 334 can appear if the communication connectionexhibits high latency, high error rate, or high rate of packet loss,where the degree that qualifies as “high” can be a degree that exceeds apredetermined threshold.

The layout and configuration of the visual display of FIG. 14 is merelyexemplary, and variations are contemplated to be within the scope of thepresent disclosure. For example, where the autonomous vehicle is a railvehicle, the visual interface of the remote human operator system 300can be tailored to the visual elements for operating such a vehicle.

With continuing reference to FIG. 14, in various embodiments, the visualdisplay 330 of the human operator system 300 can highlight objects onthe visual display relating to a potential hazard condition, such ashighlighting traffic lights or road signs or displaying them asgraphical icons. In various embodiments, the visual display 330 of thehuman operator system 300 highlights other potential hazard conditions,such as surrounding objects such as vehicles, pedestrians, cyclists,obstacles, lights, signs, lane lines, turning lanes, and curbs, amongother things. These features can be dynamically performed by theautonomous vehicle and/or the remote human operator system 300 based ondetected objects. In various embodiments, the autonomous vehicle and/orthe remote human operator system 300 can determine a recommended patharound surrounding objects and can provide indications to the humanoperator as a guide for the recommended path. In various embodiments,the indications can include haptic feedback through steering wheel ofthe remote human operator system and/or a projected path displayed ontothe visual display 330.

Other variations are contemplated to be within the scope of the presentdisclosure. For example, in various embodiments, the visual display 330of the human operator system 300 can display the speed limit. In variousembodiments, the visual display 330 of the human operator system 300 candisplay a tailgater and/or rear collision warning. For example, thewarning can be displayed when a tailgating vehicle is following tooclosely based on its vehicle speed and the speed of the autonomousvehicle, and/or when the remote human operator applies the brakes andthere is a following vehicle or tailgater that could cause a rearcollision.

In various embodiments, the visual display 330 of the human operatorsystem 300 can display a warning message 334 when poor communicationconnections or conditions are detected. In various embodiments, thevisual display 330 of the human operator system 300 can display awarning message 334 when an unusual or anomalous situation is detected,such as, for example, an accident, police directing traffic, roadclosure, or parade, among other things.

In various embodiments, the visual display 330 of the human operatorsystem 300 can display navigation directions (not shown), such asdirectional arrows overlaid onto the displayed environment of theautonomous vehicle. In various embodiments, the navigation directionscan be two-dimensional pictorial representations of the roadwaysdisplayed in a particular region of the visual display 330, such astop-left region of the visual display 330 or another region.

Referring again to FIG. 12, an in accordance with aspects of the presentdisclosure, an autonomous vehicle can be configured to determine when aremote human operator should be requested based on captured information.As described above herein, the sensor system 170 can capture informationabout the surrounding environment of the autonomous vehicle. Forexample, the sensor system 170 can include a high frame-rate camera thatcaptures images and/or videos of the surrounding environment. Otherinformation can be captured, such as LiDAR information or other sensorinformation. In various embodiments, the sensor system 170 can captureinformation about the functioning of vehicle systems and modules, suchas proper functioning of the navigation system or power system. Forexample, the navigation system can be tested based on GPS functionality,and the power system can be tested based on voltage sensors. Theprocessor 125 can analyze the captured information and determine whetherthere is any condition or situation that would warrant requesting aremote human operator. If the processor 125 determines that a remotehuman operator should be requested, the autonomous vehicle makes arequest to a remote human operator management system via thecommunication module 160. The remote human operator system will bedescribed in more detail herein in connection with FIG. 17.

FIG. 16 shows a flow diagram of an exemplary operation of an autonomousvehicle for requesting a remote human operator. At step 502, theautonomous vehicle captures sensor information. In various embodiments,the captured sensor information can include surrounding environmentinformation and/or vehicle subsystem information. At step 504, theautonomous vehicle decides, based on the captured sensor information, torequest a remote human operator. In various embodiments, the autonomousvehicle can decide to request a remote human operator based on a vehiclesubsystem not functioning properly, such as the navigation subsystem. Invarious embodiments, the autonomous vehicle can decide to request aremote human operator based on not recognizing the surroundingenvironment situation. For example, the autonomous vehicle may notrecognize an object in the surrounding environment, such as a temporaryroad block, and may request a remote human operator to assume control ofthe vehicle. In various embodiments, the autonomous vehicle mayrecognize the surrounding environment and decide that the surroundingenvironment situation should not be handled autonomously. For example,the autonomous vehicle may decide to request a remote human operatorwhen it determines that an emergency services vehicle is signaling anemergency, a human is directing traffic, there is construction, trafficpattern has been modified or disrupted, weather conditions are unsafe,and/or there is an excessive number of pedestrians or vehicles, amongother situations. Persons skilled in the art will understand techniquesfor identifying objects in the surrounding environment of an autonomousvehicle, including machine learning techniques. In various embodiments,an excessive number of pedestrians or vehicles can be determined basedon the number of pedestrians or vehicles exceeding a threshold number.

In various embodiments, the autonomous vehicle may decide to request aremote human operator when it cannot autonomously determine the nextmaneuver to select or execute. For example, the autonomous vehicle mayrecognize all objects and situations, but may be unable to resolve whichmaneuver to execute next. Some situations may include, for example,particularly difficult construction areas or inclement weatherconditions, among other situations.

In various embodiments, the autonomous vehicle may decide to request aremote human operator when it encounters a predetermined situation or apredetermined roadway. For example, the predetermined situations or thepredetermined roadways may include particularly difficult roads,particular types of turns, or presence of an emergency vehicle, amongother situations or roadways. In various embodiments, the autonomousvehicle may decide to request a remote human operator when it does notrecognize a situation involving a subsystem of the vehicle. In variousembodiments, the autonomous vehicle may decide to request a remote humanoperator when it travels outside a predetermined operating geography.For example, accidents or construction may force the vehicle to takedetours and travel outside a designated operating geography. In variousembodiments, the autonomous vehicle may decide to request a remote humanoperator when multiple attempts to execute an autonomous maneuver havefailed. For example, such a situation may occur if the vehicle attemptsto make an unprotected left turn but has not done so after a certainnumber of minutes. The situations described above are exemplary, andother situations are contemplated for requesting a remote humanoperator.

At step 506, the autonomous vehicle determines a risk rating based onthe captured information. The risk rating can depend on the particularsituation reflected by the captured information. In various embodiments,the risk rating can include three levels: a low risk rating, a high riskrating, and a moderate risk rating. In various embodiments, the riskrating can include two levels or more than three levels.

In various embodiments, the autonomous vehicle can determine a riskrating based on how much time may be available to safely react to thesituation. The autonomous vehicle can access risk mitigation procedures,estimate the probability of success in executing the risk mitigationprocedures, and determine the risk rating based on the estimatedprobability of success. In various embodiments, the risk mitigationprocedures can include finding a spot to pull over, finding an easier orslower street to travel on while waiting for a remote human operator, orstopping on a roadway without interfering with traffic, among otherprocedures. In various embodiments, estimating the probability ofsuccess in executing the risk mitigation procedure can be based on oneor more factors, such as, for example, road speed limits, currentvehicle speed, current speed of surrounding vehicles, number ofsurrounding vehicles, number of surrounding pedestrians, number ofsurrounding objects, road width, weather conditions, available time toreact, and/or proximity to surrounding vehicles. In various embodiments,other factors can be considered. In various embodiments, a lowerprobability of success can correspond to a higher risk rating, and ahigher probability of success can correspond to a lower risk rating.Accordingly, the autonomous vehicle can determine the risk ratingdynamically and multi-dimensionally.

As an example, where an emergency vehicle attempts to pass by and theautonomous vehicle is on a slow road with a clear space to pull over,the risk rating can be determined to be a low risk rating. As anotherexample, where a sub-system of the autonomous vehicle malfunctions on ahigh speed road without room to pull over, the risk rating can bedetermined to be a highest risk.

At step 508, the autonomous vehicle communicates the request to theremote human operator management system for a remote human operator toassume control. The request includes the risk rating and at least aportion of the captured information. For example, if the capturedinformation contains image or video information, at least some of theimages or a portion of the video can be communicated to the remote humanoperator management system. The remote human operator management system,which will be described in more detail in connection with FIG. 17, canutilize risk rating and/or the captured information in the request toassign a remote human operator to the request. In various embodiments,the request can include further information, such as locationinformation and/or information indicating the particular situationprompting the request for a remote human operator. In variousembodiments, the request can include information indicating that avehicle subsystem is not functioning properly, or information indicatingthat an emergency services vehicle is signaling an emergency, a human isdirecting traffic, and/or there is an excessive number of pedestrians orvehicles, among other situations.

At step 510, the autonomous vehicle can control the conveyance system tomitigate risk while waiting for a remote human operator to assumecontrol. In various embodiments, the autonomous vehicle can evaluatedifferent ways to mitigate risk, including parking at a curb, parking ina parking lot, traveling on roadways in search of parking, traveling ata slower speed than the travel speed without the risk; or departing froma planned route to a roadway having lower risk. Other ways to mitigaterisk are contemplated, such as stopping completely in place or followingother vehicles until a remote human operator assumes control. Theautonomous vehicle can determine and carry out the most effective optionto mitigate risk for the situation. For example, if the situationinvolves the navigation system not functioning properly, the autonomousvehicle can park at a curb or travel on roadways in search of parking.If the situation involves an emergency services vehicle signaling anemergency, the autonomous vehicle can follow the vehicle in front of it.If the situation involves a human directing traffic, the autonomousvehicle can travel at a slow speed. If the situation involves anexcessive number of pedestrians or vehicles, the autonomous vehicle canstop in place. The risk mitigation options and choices described aboveare exemplary, and other risk mitigation options and choices arecontemplated.

Referring again to step 506, in various embodiments, the autonomousvehicle can determine the risk rating based also on the available riskmitigation options. For example, if there is at least one riskmitigation option that can be safely performed, the risk rating can bedecreased. On the other hand, if there is no risk mitigation option thatcan be safely performed, the risk rating can be increased.

The operation of FIG. 16 is exemplary, and other implementations arecontemplated for requesting a remote human operator and for mitigatingrisk while waiting for a remote human operator to assume control.

Referring now to FIG. 17, there is shown a flow diagram of an exemplaryoperation of a remote human operator management system. The remote humanoperator management system communicates with autonomous vehicles andremote human operator systems (300, FIG. 13), and operates to assignremote human operators to autonomous vehicles that request a remotehuman operator. In accordance with aspects of the present disclosure,the remote human operator management system includes or has access to adatabase of experience information on remote human operators. Theexperience information can include information such as a remote humanoperator's experience driving in particular regions, such as particularstates, cities, towns, or neighborhoods, and/or experience handlingparticular situations, such as when a vehicle subsystem is notfunctioning properly, when an emergency vehicle is signaling anemergency, when a human is directing traffic, and/or when there is anexcessive number of pedestrians or vehicles, among other situations.

In various embodiments, and with reference also to FIG. 11, the remotehuman operator management system and/or the database can be part of thecentral server 110 and/or the fleet management module 120. In variousembodiments, the remote human operator management system and/or thedatabase can be separate from the central server 110 and the fleetmanagement module 120.

With continuing reference to FIG. 17, at step 602, the remote humanoperator management system receives requests from remote autonomous landvehicles for remote human operators to assume control. As describedabove, each request can include a risk rating generated by thecorresponding remote autonomous land vehicle and can include otherinformation such as information captured by the sensor system, locationinformation, and/or information indicating a particular situation. Invarious embodiments, the requests may not include a risk rating but caninclude risk information generated by the autonomous vehicle. In suchsituations, the remote human operator management system can determine arisk rating based on the risk information generated by the autonomousvehicle. At step 604, the remote human operator management systemdetermines that the number of requests from autonomous vehicles for aremote human operator exceeds the number of available remote humanoperators. At step 606, the remote human operator management systemassigns at least some of the available remote human operators to atleast some of the remote autonomous land vehicles based on the riskratings and/or the experience information in the database. In variousembodiments, the remote human operator management system prioritizes theassignment of available remote human operators to autonomous vehicleshaving the highest risk rating. In various embodiments, the remote humanoperator management system can assign remote human operators who havemore experience to the highest risk ratings. In various embodiments, theremote human operator management system can assign available remotehuman operators who have experience driving in particular regions toremote autonomous vehicles located in those regions. In variousembodiments, the remote human operator management system can assignavailable remote human operators who have experience handling vehiclesubsystem failures to remote autonomous vehicles experience subsystemfailures. In various embodiments, the remote human operator managementsystem can assign available remote human operators who have experiencewith particular situations to remote autonomous vehicles facing suchsituations. For example, the situations can include an emergency vehiclesignaling an emergency, a human directing traffic, and/or an excessivenumber of pedestrians or vehicles. Such experience information can bestored in the database, as described above.

The operation of FIG. 17 is exemplary and variations are contemplated tobe within the scope of the present disclosure. For example, in variousembodiments, the remote human operator management system can determinethat the number of available remote human operators is greater than thenumber of requests from autonomous vehicles for remote human operators.In that situation, the remote human operator management system maypresent a list of requests to each remote human operator and permit theremote human operator to select a request from the list. The list can bepresented on the visual interface 330 of the remote human operatorsystem 300 (FIG. 13.) In various embodiments, the requests presented toeach remote human operator can be based on the experience information inthe database. In various embodiments, the requests presented to eachremote human operator can be based on new situations that a remote humanoperator would like to gain experience handling. The remote humanoperator can indicate preferences for such new experiences, and thepreferences can be stored in the database.

It is contemplated that the embodiments disclosed in connection withFIGS. 13-17 can be combined in various ways. Accordingly, theillustrated and described embodiments are merely exemplary and do notlimit the scope of the present disclosure.

Additional Features

In some embodiments, the robot fleet further includes at least one robothaving a digital display for curated content comprising: advertisements(i.e., for both specific user and general public), including servicesprovided, marketing/promotion, regional/location of areas served,customer details, local environment, lost, sought or detected people,public service announcements, date, time, or weather.

The embodiments disclosed herein are examples of the disclosure and maybe embodied in various forms. For instance, although certain embodimentsherein are described as separate embodiments, each of the embodimentsherein may be combined with one or more of the other embodiments herein.Specific structural and functional details disclosed herein are not tobe interpreted as limiting, but as a basis for the claims and as arepresentative basis for teaching one skilled in the art to variouslyemploy the present disclosure in virtually any appropriately detailedstructure. Like reference numerals may refer to similar or identicalelements throughout the description of the figures.

The phrases “in an embodiment,” “in embodiments,” “in variousembodiments,” “in some embodiments,” or “in other embodiments” may eachrefer to one or more of the same or different embodiments in accordancewith the present disclosure. A phrase in the form “A or B” means “(A),(B), or (A and B).” A phrase in the form “at least one of A, B, or C”means “(A); (B); (C); (A and B); (A and C); (B and C); or (A, B, andC).”

Any of the herein described methods, programs, algorithms or codes maybe converted to, or expressed in, a programming language or computerprogram. The terms “programming language” and “computer program,” asused herein, each include any language used to specify instructions to acomputer, and include (but is not limited to) the following languagesand their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++,Delphi, Fortran, Java, JavaScript, machine code, operating systemcommand languages, Pascal, Perl, PL1, scripting languages, Visual Basic,metalanguages which themselves specify programs, and all first, second,third, fourth, fifth, or further generation computer languages. Alsoincluded are database and other data schemas, and any othermeta-languages. No distinction is made between languages which areinterpreted, compiled, or use both compiled and interpreted approaches.No distinction is made between compiled and source versions of aprogram. Thus, reference to a program, where the programming languagecould exist in more than one state (such as source, compiled, object, orlinked) is a reference to any and all such states. Reference to aprogram may encompass the actual instructions and/or the intent of thoseinstructions.

The systems described herein may also utilize one or more controllers toreceive various information and transform the received information togenerate an output. The controller may include any type of computingdevice, computational circuit, or any type of processor or processingcircuit capable of executing a series of instructions that are stored ina memory. The controller may include multiple processors and/ormulticore central processing units (CPUs) and may include any type ofprocessor, such as a microprocessor, digital signal processor,microcontroller, programmable logic device (PLD), field programmablegate array (FPGA), or the like. The controller may also include a memoryto store data and/or instructions that, when executed by the one or moreprocessors, causes the one or more processors to perform one or moremethods and/or algorithms.

Any of the herein described methods, programs, algorithms or codes maybe converted to, or expressed in, a programming language or computerprogram. The terms “programming language” and “computer program,” asused herein, each include any language used to specify instructions to acomputer, and include (but is not limited to) the following languagesand their derivatives: Assembler, Basic, Batch files, BCPL, C, C+, C++,Delphi, Fortran, Java, JavaScript, machine code, operating systemcommand languages, Pascal, Perl, PL1, scripting languages, Visual Basic,metalanguages which themselves specify programs, and all first, second,third, fourth, fifth, or further generation computer languages. Alsoincluded are database and other data schemas, and any othermeta-languages. No distinction is made between languages which areinterpreted, compiled, or use both compiled and interpreted approaches.No distinction is made between compiled and source versions of aprogram. Thus, reference to a program, where the programming languagecould exist in more than one state (such as source, compiled, object, orlinked) is a reference to any and all such states. Reference to aprogram may encompass the actual instructions and/or the intent of thoseinstructions.

It should be understood that the foregoing description is onlyillustrative of the present disclosure. Various alternatives andmodifications can be devised by those skilled in the art withoutdeparting from the disclosure. Accordingly, the present disclosure isintended to embrace all such alternatives, modifications and variances.The embodiments described with reference to the attached drawing figuresare presented only to demonstrate certain examples of the disclosure.Other elements, steps, methods, and techniques that are insubstantiallydifferent from those described above and/or in the appended claims arealso intended to be within the scope of the disclosure.

What is claimed is:
 1. An autonomous robot land vehicle comprising: aland vehicle conveyance system; a sensor system configured to captureinformation including at least one of: surrounding environmentinformation or vehicle subsystem information; a communication systemconfigured to communicate with a remote human operator managementsystem; at least one processor; and a memory storing instructions which,when executed by the at least one processor, cause the autonomous robotland vehicle to, autonomously: determine, based on the capturedinformation, to request a remote human operator, and communicate arequest to the remote human operator management system for a remotehuman operator to assume control of the land vehicle conveyance system,the request including at least a portion of the captured information. 2.The autonomous robot land vehicle of claim 1, wherein in determining torequest a remote human operator, the instructions, when executed by theat least one processor, cause the autonomous robot land vehicle todetermine, based on the vehicle subsystem information, that at least onesubsystem is not functioning properly.
 3. The autonomous robot landvehicle of claim 1, wherein in determining to request a remote humanoperator, the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to determine, based on thecaptured information, that a next maneuver cannot be autonomouslydetermined.
 4. The autonomous robot land vehicle of claim 1, wherein indetermining to request a remote human operator, the instructions, whenexecuted by the at least one processor, cause the autonomous robot landvehicle to determine, based on the surrounding environment information,that a surrounding environment situation is not recognized.
 5. Theautonomous robot land vehicle of claim 4, wherein in determining torequest a remote human operator, the instructions, when executed by theat least one processor, cause the autonomous robot land vehicle to:access predetermined situations or predetermined roadways that arepredetermined to require a request for a remote human operator; anddetermine, based on the captured information, that at least one of thepredetermined situations or the predetermined roadways has beenencountered.
 6. The autonomous robot land vehicle of claim 1, wherein indetermining to request a remote human operator, the instructions, whenexecuted by the at least one processor, cause the autonomous robot landvehicle to determine, based on the surrounding environment information,that a surrounding environment situation should not be autonomouslyhandled.
 7. The autonomous robot land vehicle of claim 6, wherein indetermining to request a remote human operator, the instructions, whenexecuted by the at least one processor, cause the autonomous robot landvehicle to determine, based on the vehicle subsystem information, that asituation of at least one subsystem is not recognized.
 8. The autonomousrobot land vehicle of claim 6, wherein in determining to request aremote human operator, the instructions, when executed by the at leastone processor, cause the autonomous robot land vehicle to: access apredetermined operating geography; and determine, based on thesurrounding environment information, that the autonomous robot landvehicle is outside the predetermined operating geography.
 9. Theautonomous robot land vehicle of claim 6, wherein in determining torequest a remote human operator, the instructions, when executed by theat least one processor, cause the autonomous robot land vehicle todetermine, based on the captured information, that multiple attempts toexecute an autonomous maneuver have failed.
 10. The autonomous robotland vehicle of claim 1, wherein the instructions, when executed by theat least one processor, cause the autonomous robot land vehicle todetermine a risk rating based on the captured information.
 11. Theautonomous robot land vehicle of claim 10, wherein in determining a riskrating based on the captured information, the instructions, whenexecuted by the at least one processor, cause the autonomous robot landvehicle to determine the risk rating based on estimated safety ofautonomous risk mitigation procedures that can be effectuated whilewaiting for a remote human operator to assume control.
 12. Theautonomous robot land vehicle of claim 11, wherein in determining therisk rating based on estimated safety of autonomous risk mitigationprocedures, the instructions, when executed by the at least oneprocessor, cause the autonomous robot land vehicle to: access the riskmitigation procedures; estimate, based on the captured information,probability of success in executing the risk mitigation procedures; anddetermine the risk rating based on the estimated probability of success.13. The autonomous robot land vehicle of claim 12, wherein in estimatingthe probability of success in executing the risk mitigation procedures,the instructions, when executed by the at least one processor, cause theautonomous robot land vehicle to estimate the probability of successbased on at least one of: road speed limits, current vehicle speed,current speed of surrounding vehicles, number of surrounding vehicles,number of surrounding pedestrians, number of surrounding objects, roadwidth, weather conditions, available time to react, or proximity tosurrounding vehicles.
 14. The autonomous robot land vehicle of claim 10,wherein the instructions, when executed by the at least one processor,cause the autonomous robot land vehicle to control the land vehicleconveyance system to mitigate risk while waiting for a remote humanoperator to assume control.
 15. The autonomous robot land vehicle ofclaim 14, wherein in controlling the land vehicle conveyance system tomitigate risk, the instructions, when executed by the at least oneprocessor, cause the autonomous robot land vehicle to perform at leastone of: park at a curb, parking in a parking lot, travel on roadways insearch of parking, or depart from a planned route to a roadway havinglower risk.
 16. The autonomous robot land vehicle of claim 14, whereinin controlling the land vehicle conveyance system to mitigate risk, theinstructions, when executed by the at least one processor, cause theautonomous robot land vehicle to travel at a slower speed than a travelspeed without the risk.
 17. The autonomous robot land vehicle of claim1, wherein the autonomous robot land vehicle does not contain any spacefor a human passenger or human operator to be within the autonomousrobot land vehicle.
 18. A system for assigning remote human operators toautonomous land vehicles, the system comprising: a database includingexperience information on a plurality of remote human operators; acommunication system configured to communicate with a plurality ofremote autonomous land vehicles; at least one processor; and a memorystoring instructions which, when executed by the at least one processor,cause the system to: receive, via the communication system, requestsfrom the remote autonomous land vehicles for remote human operators toassume control, each of the requests including risk informationgenerated by the corresponding remote autonomous land vehicle, determinethat a number of the requests is greater than a number of availableremote human operators, access risk ratings associated with therequests, wherein the risk ratings include at least one of: ratingsdetermined by the system based on the risk information generated by theremote autonomous land vehicles or ratings that are determined by theremote autonomous land vehicles and included in the risk informationgenerated by the remote autonomous land vehicles; and assign at leastsome of the available remote human operators to at least some of theremote autonomous land vehicles based on at least one of: the riskratings or the experience information on the plurality of remote humanoperators.
 19. The system of claim 18, wherein in assigning at leastsome of the available remote human operators to at least some of theremote autonomous land vehicles, the instructions, when executed by theat least one processor, cause the system to prioritize assignment of theavailable remote human operators to remote autonomous land vehicleshaving highest risk ratings.
 20. The system of claim 19, wherein theexperience information includes an amount of experience, and wherein inprioritizing assignment of the available remote human operators toremote autonomous land vehicles having highest risk ratings, theinstructions, when executed by the at least one processor, cause thesystem to prefer available remote human operators who have higheramounts of experience.
 21. The system of claim 18, wherein theexperience information includes experience driving in particularregions, and wherein in assigning at least some of the available remotehuman operators to at least some of the remote autonomous land vehicles,the instructions, when executed by the at least one processor, cause thesystem to: determine, based on the experience information, a group ofthe available remote human operators who have experience driving in aparticular region in which one of the remote autonomous land vehicles islocated; and assign one of the remote human operators from the group tothe one of the remote autonomous land vehicles.
 22. The system of claim18, wherein the experience information includes experience driving inparticular situations, and wherein each of the requests includes anindication of a risk situation.
 23. The system of claim 22, wherein therisk situation includes a vehicle subsystem not functioning properly,and wherein in assigning at least some of the available remote humanoperators to at least some of the remote autonomous land vehicles, theinstructions, when executed by the at least one processor, cause thesystem to: determine, based on the experience information, a group ofthe available remote human operators who have experience remotelyoperating a vehicle with the vehicle subsystem not functioning properly;and assign one of the remote human operators from the group to the oneof the remote autonomous land vehicles.
 24. The system of claim 23,wherein vehicle subsystem that is not functioning properly is anavigation subsystem.
 25. The system of claim 22, wherein the risksituation includes a surrounding environment situation, and wherein inassigning at least some of the available remote human operators to atleast some of the remote autonomous land vehicles, the instructions,when executed by the at least one processor, cause the system to:determine, based on the experience information, a group of the availableremote human operators who have experience remotely operating a vehiclein the surrounding environment situation; and assign one of the remotehuman operators from the group to the one of the remote autonomous landvehicles.
 26. The system of claim 25, wherein the surroundingenvironment situation includes at least one of: an unrecognized object,an emergency services vehicle signaling an emergency, a human directingtraffic, or an excessive number of pedestrians or vehicles.